feat: create pod with sample app and model services after build is completed (#90) #100

lstocchi · 2024-01-19T15:42:03Z

This PR adds an additional step to the pullApplication action - it creates a pod with the 2 containers.
It doesn't get started yet, i'm going to work on it on a follow-up PR but it can be start manually from the pod/container page of desktop.

EVentually you should see the green check on create application

and a new pod similar to

If you start it you would probably see it fail because the modelservice is slow at loading the model and the sample app when trying to connect to the model service it fails.
So what you can do is

start the pod
the model service container should appear green
check the logs and wait until you see it is completely loaded and running (e.g there should be http://localhost:7860 url message)
start the sample app container
now you can open it on the browser

video demo -> https://drive.google.com/file/d/16hWByy2J17s88o_O6WyFcCwPAVczH3E8/view

feloy · 2024-01-19T16:02:56Z

When testing on my machine (Fedora Linux), the image is correctly built and the container is started, but the container stops after a few seconds.
I cannot debug the container started by the AI Studio, but here is what I get when I run the container from the CLI (do I need to add some options to the run command?):

podman run -it chatbot-example  bash
Loaded as API: http://10.88.0.1:7860/ ✔
Traceback (most recent call last):
  File "/opt/app-root/lib64/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/opt/app-root/lib64/python3.9/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
    sock = socket.create_connection(
  File "/usr/lib64/python3.9/socket.py", line 844, in create_connection
    raise err
  File "/usr/lib64/python3.9/socket.py", line 832, in create_connection
    sock.connect(sa)
socket.timeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/app-root/lib64/python3.9/site-packages/httpx/_transports/default.py", line 67, in map_httpcore_exceptions
    yield
  File "/opt/app-root/lib64/python3.9/site-packages/httpx/_transports/default.py", line 231, in handle_request
    resp = self._pool.handle_request(req)
  File "/opt/app-root/lib64/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
    raise exc
  File "/opt/app-root/lib64/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
    response = connection.handle_request(request)
  File "/opt/app-root/lib64/python3.9/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/opt/app-root/lib64/python3.9/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
  File "/opt/app-root/lib64/python3.9/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
  File "/opt/app-root/lib64/python3.9/site-packages/httpcore/_backends/sync.py", line 213, in connect_tcp
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/app-root/lib64/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectTimeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/chatbot/chat_ui.py", line 22, in <module>
    chat = Chat(model_endpoint)
  File "/chatbot/chat_ui.py", line 9, in __init__
    self.client = Client(self.endpoint)
  File "/opt/app-root/lib64/python3.9/site-packages/gradio_client/client.py", line 133, in __init__
    self.config = self._get_config()
  File "/opt/app-root/lib64/python3.9/site-packages/gradio_client/client.py", line 720, in _get_config
    r = httpx.get(
  File "/opt/app-root/lib64/python3.9/site-packages/httpx/_api.py", line 196, in get
    return request(
  File "/opt/app-root/lib64/python3.9/site-packages/httpx/_api.py", line 104, in request
    return client.request(
  File "/opt/app-root/lib64/python3.9/site-packages/httpx/_client.py", line 828, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/opt/app-root/lib64/python3.9/site-packages/httpx/_client.py", line 915, in send
    response = self._send_handling_auth(
  File "/opt/app-root/lib64/python3.9/site-packages/httpx/_client.py", line 943, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/opt/app-root/lib64/python3.9/site-packages/httpx/_client.py", line 980, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/opt/app-root/lib64/python3.9/site-packages/httpx/_client.py", line 1016, in _send_single_request
    response = transport.handle_request(request)
  File "/opt/app-root/lib64/python3.9/site-packages/httpx/_transports/default.py", line 231, in handle_request
    resp = self._pool.handle_request(req)
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/app-root/lib64/python3.9/site-packages/httpx/_transports/default.py", line 84, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: timed out

lstocchi · 2024-01-19T16:21:07Z

@feloy yes it's in draft for that reason 😆
It only works for the model services app so far. It takes a while as its slow to load the model (atleast on windows) but you can see it succeed on the logs and then open it on the browser. I was waiting for Jeff as i need to ask him how i should connect the sample app to the model. I use my personal ip to connect to it but i'm sure there should be some other way.

lstocchi · 2024-01-24T09:32:16Z

I have to write some tests but you can start playing with it and see if you face some issue.
I only tested on Windows so far. It extremely slow at starting the model_service. Like 4 mins this morning :/
So if you start the pod, be patient and don't give up 😆
You can check the video above to see how it works.
I have updated https://github.com/axel7083/locallm so if you have a cached chatbot repo, delete it.

feloy · 2024-01-24T11:57:19Z

Works great on Linux Fedora. The model starts instantly on my system (not fast enough for the app to get connected the first time, a retry on the connection to the model would be helpful)

lstocchi · 2024-01-24T12:13:17Z

Works great on Linux Fedora. The model starts instantly on my system (not fast enough for the app to get connected the first time, a retry on the connection to the model would be helpful)

Oh nice, so windows is the only turtle here 😆
Yes in a following PR i think we need to listen to the port to see when the model service is ready and then connect the sample app 👍

feloy · 2024-01-24T13:09:24Z

Works great on Linux Fedora. The model starts instantly on my system (not fast enough for the app to get connected the first time, a retry on the connection to the model would be helpful)

Oh nice, so windows is the only turtle here 😆 Yes in a following PR i think we need to listen to the port to see when the model service is ready and then connect the sample app 👍

I created a PR: https://github.com/axel7083/locallm/pull/4

lstocchi · 2024-01-24T16:39:30Z

@feloy @axel7083 @jeffmaury i've splitted the commits so the first one is just a refactoring of the existing code (instead of having a hube pullApplication i split it into smaller pieces), the second is about new stuff and then 3 and 4 all the tests.
If you want you can look at each commit separately to have a cleaner view. Just FYI.

jeffmaury · 2024-01-25T07:51:39Z

packages/backend/src/managers/applicationManager.ts

    });
  }

-  private downloadModel(
+  private doDownloadModel(


Should be in modelsManager

Agree. I'll move it in another PR 👍

lstocchi · 2024-01-25T08:41:47Z

I'll wait another hour so someone else have the time to give it a look and i'll merge as this is the base for moving forward.

feloy

great work, thanks

Signed-off-by: lstocchi <[email protected]>

update audio-to-text for ai studio

lstocchi changed the title ~~feat: create/run sample app containers after image built completed (#90)~~ feat: create pod with sample app and model services after build is completed (#90) Jan 23, 2024

lstocchi force-pushed the i90 branch from 276cfc2 to a6e0efa Compare January 24, 2024 08:29

lstocchi force-pushed the i90 branch 4 times, most recently from 39b3860 to 8ddb6f8 Compare January 24, 2024 11:46

lstocchi force-pushed the i90 branch from 8ddb6f8 to 4d73baa Compare January 24, 2024 16:32

lstocchi marked this pull request as ready for review January 24, 2024 16:37

lstocchi requested review from feloy, axel7083 and jeffmaury January 24, 2024 16:37

jeffmaury approved these changes Jan 25, 2024

View reviewed changes

This was referenced Jan 25, 2024

Start the pod with sample app and model service #129

Closed

Model service not running on MAC applehv and linux #130

Closed

Create the containers inside the pod #131

Closed

doDownloadModel should be in modelsManager #133

Closed

feloy approved these changes Jan 25, 2024

View reviewed changes

lstocchi added 5 commits January 25, 2024 10:38

fix: refactor pullApplication

072cef4

Signed-off-by: lstocchi <[email protected]>

feat: add step that create the pod with the containers

fb65293

Signed-off-by: lstocchi <[email protected]>

fix: fix failing test, format, lint

de165fc

Signed-off-by: lstocchi <[email protected]>

fix: add tests

ee91d52

Signed-off-by: lstocchi <[email protected]>

fix: fix after rebase

702cbda

Signed-off-by: lstocchi <[email protected]>

lstocchi force-pushed the i90 branch from 4d73baa to 702cbda Compare January 25, 2024 09:46

lstocchi enabled auto-merge (squash) January 25, 2024 09:47

lstocchi merged commit 6c5ca63 into main Jan 25, 2024
3 checks passed

lstocchi deleted the i90 branch January 25, 2024 09:48

lstocchi mentioned this pull request Jan 25, 2024

Mount volume when starting model service app #90

Closed

mhdawson pushed a commit to mhdawson/podman-desktop-extension-ai-lab that referenced this pull request Nov 22, 2024

Merge pull request containers#100 from MichaelClifford/whisper

bd7eace

update audio-to-text for ai studio

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: create pod with sample app and model services after build is completed (#90) #100

feat: create pod with sample app and model services after build is completed (#90) #100

lstocchi commented Jan 19, 2024 •

edited

Loading

feloy commented Jan 19, 2024

lstocchi commented Jan 19, 2024

lstocchi commented Jan 24, 2024 •

edited

Loading

feloy commented Jan 24, 2024

lstocchi commented Jan 24, 2024

feloy commented Jan 24, 2024

lstocchi commented Jan 24, 2024 •

edited

Loading

jeffmaury Jan 25, 2024

lstocchi Jan 25, 2024

lstocchi commented Jan 25, 2024

feloy left a comment

feat: create pod with sample app and model services after build is completed (#90) #100

feat: create pod with sample app and model services after build is completed (#90) #100

Conversation

lstocchi commented Jan 19, 2024 • edited Loading

feloy commented Jan 19, 2024

lstocchi commented Jan 19, 2024

lstocchi commented Jan 24, 2024 • edited Loading

feloy commented Jan 24, 2024

lstocchi commented Jan 24, 2024

feloy commented Jan 24, 2024

lstocchi commented Jan 24, 2024 • edited Loading

jeffmaury Jan 25, 2024

Choose a reason for hiding this comment

lstocchi Jan 25, 2024

Choose a reason for hiding this comment

lstocchi commented Jan 25, 2024

feloy left a comment

Choose a reason for hiding this comment

lstocchi commented Jan 19, 2024 •

edited

Loading

lstocchi commented Jan 24, 2024 •

edited

Loading

lstocchi commented Jan 24, 2024 •

edited

Loading